Identifying optimal weights to improve prediction accuracy in machine learning techniques

ABSTRACT

A computer-implemented method, system and computer program product for improving prediction accuracy in machine learning techniques. A teacher model is constructed, where the teacher model generates a weight for each data case. The current student model is then trained using training data and the weights generated by the teacher model. After training the current student model, the current student model generates state features, which are used by the teacher model to generate new weights. A candidate student model is then trained using training data and these new weights. A reward is generated by comparing the current student model with the candidate student model using training and testing data, which is used to update the teacher model if a stopping rule has not been satisfied. Upon a stopping rule being satisfied, the weights generated by the teacher model are deemed to be the “optimal” weights which are returned to the user.

TECHNICAL FIELD

The present invention relates generally to predictive modeling, and moreparticularly to identifying optimal weights to improve predictionaccuracy in machine learning techniques.

BACKGROUND

Predictive modeling uses statistics to predict outcomes. Most often theevent one wants to predict is in the future, but predictive modellingcan be applied to any type of unknown event, regardless of when itoccurred. For example, predictive models are often used to detect eventsand identify persons related to the events, after the events have takenplace.

In many cases, the model is chosen on the basis of detection theory totry to guess the probability of an outcome given a set amount of inputdata, for example, given an email determining how likely that it isspam.

Models can use one or more classifiers in trying to determine theprobability of a set of data belonging to another set. For example, amodel might be used to determine whether an email is spam or “ham”(non-spam).

Depending on definitional boundaries, predictive modelling is synonymouswith, or largely overlapping with, the field of machine learning, as itis more commonly referred to in academic or research and developmentcontexts. When deployed commercially, predictive modelling is oftenreferred to as predictive analytics.

Achieving better predictive models is an objective in the research andpractice of machine learning techniques. For example, ensemble methodsuse multiple learning algorithms to obtain better predictive performancethan could be obtained from any of the constituent learning algorithmsalone. Such ensemble methods include bootstrap aggregating (also calledbagging), boosting, etc.

Bootstrap aggregating is a machine learning ensemble meta-algorithmdesigned to improve the stability and accuracy of machine learningalgorithms used in statistical classification and regression. Boostingis a machine learning ensemble meta-algorithm for primarily reducingbias and also variance in supervised learning and a family of machinelearning algorithms that convert weak learners to strong ones.

In such techniques, such as boosting, the weights of wrongly classifiedcases are increased while the weights of correctly classified cases aredecreased during the modeling process. Such a strategy (heuristic) doesachieve better predictions in many cases; however, overfittingoutliers/noises is a possibility. As a result of overfittingoutliers/noises, the predictive accuracy is lessened.

Hence, the heuristic strategy of increasing the weights of wronglyclassified cases and decreasing the weights of correctly classifiedcases may not be the best strategy for improving the prediction accuracyof the model.

For example, sometimes, it may be better to increase the weights ofcorrectly classified cases because such cases contain very importantpatterns which should be learned by the machine learning algorithm. Itmay also be better to decrease the weights of wrongly classified cases,such as outlier cases, for similar reasons.

Consequently, such techniques, such as boosting, fail to identify theoptimal weights for the classified cases, and therefore, fail to achieveoptimal prediction accuracy in machine learning techniques.

SUMMARY

In one embodiment of the present invention, a computer-implementedmethod for improving prediction accuracy in machine learning techniquescomprises training a current student model using training data andweights generated by a teacher model. The method further comprisesgenerating new weights by the teacher model using state features. Themethod additionally comprises training a candidate student model usingthe training data and the new weights. Furthermore, the method comprisesgenerating a reward by comparing the current student model with thecandidate student model using the training data and testing data todetermine which is better at predicting an observed target.Additionally, the method comprises returning the new weights and thecurrent student model to a user in response to a stopping rule beingsatisfied, where the returned student model provides a prediction of theobserved target.

In another embodiment of the present invention, a computer programproduct for improving prediction accuracy in machine learningtechniques, the computer program product comprising one or more computerreadable storage mediums having program code embodied therewith, theprogram code comprising programming instructions for training a currentstudent model using training data and weights generated by a teachermodel. The program code further comprises generating new weights by theteacher model using state features. The program code additionallycomprises training a candidate student model using the training data andthe new weights. Furthermore, the program code comprises generating areward by comparing the current student model with the candidate studentmodel using the training data and testing data to determine which isbetter at predicting an observed target. Additionally, the program codecomprises returning the new weights and the current student model to auser in response to a stopping rule being satisfied, where the returnedstudent model provides a prediction of the observed target.

In a further embodiment of the present invention, a system comprises amemory for storing a computer program for improving prediction accuracyin machine learning techniques and a processor connected to the memory,where the processor is configured to execute program instructions of thecomputer program comprising training a current student model usingtraining data and weights generated by a teacher model. The programinstructions of the computer program further comprise generating newweights by the teacher model using state features. The programinstructions of the computer program additionally comprise training acandidate student model using the training data and the new weights.Furthermore, the program instructions of the computer program comprisegenerating a reward by comparing the current student model with thecandidate student model using the training data and testing data todetermine which is better at predicting an observed target.Additionally, the program instructions of the computer program comprisereturning the new weights and the current student model to a user inresponse to a stopping rule being satisfied, where the returned studentmodel provides a prediction of the observed target.

In this manner, the present invention devises a framework whichimplements the concept of “learning to teach” in the field of predictivemodeling. Such a framework includes a teacher model, which generates aweight for each data case. The training data cases, along with thegenerated weights, are used to re-train the student model. A reward isreturned by evaluating the trained student model on a held-out dataset(testing data) in terms of prediction accuracy. The teacher model thenutilizes the reward to update its parameters via policy gradientmethods, e.g., reinforcement learning. Such a process will be repeateduntil the student model achieves desired performance.

In comparison to previously used heuristic methods (e.g., boosting), theapproach of the present invention determines case weights in an optimalway. This allows one to build a better student model via basic learners,e.g., decision tree, neural network, etc., rather than using an ensemblemodel.

By using case weights as actions on the student model, any type ofmachine learner may be used as the student model given that the learnersupports case weights in training.

Furthermore, the distributions of data cases in the training data can becorrected by the generated weights in the event that the training datacomes from a biased sampling.

The foregoing has outlined rather generally the features and technicaladvantages of one or more embodiments of the present invention in orderthat the detailed description of the present invention that follows maybe better understood. Additional features and advantages of the presentinvention will be described hereinafter which may form the subject ofthe claims of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained when thefollowing detailed description is considered in conjunction with thefollowing drawings, in which:

FIG. 1 illustrates a communication system for practicing the principlesof the present invention in accordance with an embodiment of the presentinvention;

FIG. 2 illustrates an embodiment of the present invention of thehardware configuration of a predictive analytics system which isrepresentative of a hardware environment for practicing the presentinvention;

FIG. 3 is a diagram of the software components of the predictiveanalytics system used to identify the optimal weights to be used by thepredictive model for generating predictions in accordance with anembodiment of the present invention;

FIG. 4 illustrates the state features generated by the student model inaccordance with an embodiment of the present invention;

FIG. 5 illustrates the rewards generated by the reward generator inaccordance with an embodiment of the present invention; and

FIG. 6 is a flowchart of a method for improving prediction accuracy inmachine learning techniques in accordance with an embodiment of thepresent invention.

DETAILED DESCRIPTION

The present invention comprises a computer-implemented method, systemand computer program product for improving prediction accuracy inmachine learning techniques. In one embodiment of the present invention,a teacher model is constructed, where the teacher model generates aweight for each data case. A “teacher model,” as used herein, refers toa statistical model that determines the appropriate data, loss functionand hypothesis space to facilitate the learning of the student model.The current student model is then trained using training data and theweights generated by the teacher model. A “student model,” as usedherein, refers to a statistical model that is trained to provide aprediction using training data. A “current” student model, as usedherein, refers to a student model currently being trained to provide aprediction using training data. The current student model generatesstate features (e.g., data features, case weights, student modelfeatures and features to represent the combination of both the data andthe student model), which are used by the teacher model to generate newweights. A candidate student model is then trained using training dataand these new weights. A “candidate student model,” as used herein,refers to a student model that is being examined to determine if is abetter student model (better at predicting the observed target) than thecurrent student model. A reward is then generated by comparing thecurrent student model with the candidate student model using trainingand testing data to determine which is better at predicting an observedtarget. A “reward,” as used herein, refers to a value generated by afunction (reward function) used in reinforcement learning. A positivereward may be returned if the candidate student model is better atpredicting the observed target than the current student model.Conversely, a negative reward may be returned if the current studentmodel is better at predicting the observed target than the candidatestudent model. The teacher model is then updated with the reward. Theteacher model utilizes the rewards to update its parameters via policygradient methods, such as reinforcement learning. If the candidatestudent model is better at predicting the observed target than thecurrent student model, then the current student model is updated withthe candidate student model and the current weights are updated with thenew weights generated by the teacher model. Upon updating the currentweights with the new weights, the current student model generates newstate features. If, however, the candidate student model is not betterat predicting the observed target than the current student model, thenthe updated teacher model generates new weights using the currentweights and the current student features from the current student model.Upon any of the stopping rules being satisfied (e.g., reaching aspecified number of trials, reaching a specified training timing,converging of a prediction accuracy and a user-initiated termination),the weights generated by the teacher model are deemed to be the“optimal” weights which are returned to the user along with thecorresponding student model. In this manner, optimal weights areidentified to improve prediction accuracy.

In the following description, numerous specific details are set forth toprovide a thorough understanding of the present invention. However, itwill be apparent to those skilled in the art that the present inventionmay be practiced without such specific details. In other instances,well-known circuits have been shown in block diagram form in order notto obscure the present invention in unnecessary detail. For the mostpart, details considering timing considerations and the like have beenomitted inasmuch as such details are not necessary to obtain a completeunderstanding of the present invention and are within the skills ofpersons of ordinary skill in the relevant art.

Referring now to the Figures in detail, FIG. 1 illustrates acommunication system 100 for making predictions using machine learningtechniques. In one embodiment, system 100 includes a predictiveanalytics system 101 for generating predictions 102 using data, such astraining data 103 and testing data 104 (also referred to herein as“hold-out data”). A further description of predictive analytics system101 using training and testing data 103, 104 to make predictions isdiscussed further below in connection with FIGS. 3-6.

In one embodiment, predictive analytics system 101 makes predictionsabout unknown future events using many techniques from data mining,statistics, modeling, machine learning and artificial intelligence toanalyze current data to make predictions about the future.

In one embodiment, predictive analytics system 101 utilizes the conceptof “learning to teach,” which involves two intelligent agents, namely, ateacher model and a student model. The training phase contains severalepisodes of sequential interactions between the teacher model and thestudent model. Based on the state information generated by the studentmodel, the teacher model updates its teaching actions so as to refinethe machine learning problem of the student model. The student modelthen performs its learning process based on the inputs from the teachermodel and provides reward signals (e.g., the accuracy on the trainingdata) back to the teacher model afterwards. The teacher model thenutilizes such rewards to update its parameters via policy gradientmethods, which are a type of a reinforcement learning technique. Thisinteractive process is end-to-end trainable, exempt from the limitationsof human-defined heuristics. In one embodiment, the concept of “learningto teach” is implemented by devising an approach of leveraging theweights of cases (data cases) as actions for the student model. Adescription of the hardware configuration of predictive analytics system101 is provided below in connection with FIG. 2.

Referring now to FIG. 2, FIG. 2 illustrates an embodiment of the presentinvention of the hardware configuration of a predictive analytics system101 (FIG. 1) which is representative of a hardware environment forpracticing the present invention. Predictive analytics system 101 may beany type of analytics system (e.g., portable computing unit, PersonalDigital Assistant (PDA), laptop computer, mobile device, tablet personalcomputer, smartphone, mobile phone, navigation device, gaming unit,desktop computer system, workstation, Internet appliance and the like)configured with the capability of identifying optimal weights to improveprediction accuracy in machine learning techniques.

Referring to FIG. 2, predictive analytics system 101 may have aprocessor 201 connected to various other components by system bus 202.An operating system 203 may run on processor 201 and provide control andcoordinate the functions of the various components of FIG. 2. Anapplication 204 in accordance with the principles of the presentinvention may run in conjunction with operating system 203 and providecalls to operating system 203 where the calls implement the variousfunctions or services to be performed by application 204. Application204 may include, for example, a program for identifying optimal weightsto improve prediction accuracy in machine learning techniques asdiscussed below in connection with FIGS. 3-6.

Referring again to FIG. 2, read-only memory (“ROM”) 205 may be connectedto system bus 202 and include a basic input/output system (“BIOS”) thatcontrols certain basic functions of predictive analytics system 101.Random access memory (“RAM”) 206 and disk adapter 207 may also beconnected to system bus 202. It should be noted that software componentsincluding operating system 203 and application 204 may be loaded intoRAM 206, which may be predictive analytics system's 101 main memory forexecution. Disk adapter 207 may be an integrated drive electronics(“IDE”) adapter that communicates with a disk unit 208, e.g., diskdrive. It is noted that the program for identifying optimal weights toimprove prediction accuracy in machine learning techniques, as discussedbelow in connection with FIGS. 3-6, may reside in disk unit 208 or inapplication 204.

Predictive analytics system 101 may further include a communicationsadapter 209 connected to bus 202. Communications adapter 209 mayinterconnect bus 202 with an outside network thereby allowing predictiveanalytics system 101 to communicate with other devices.

I/O devices may also be connected to predictive analytics system 101 viaa user interface adapter 210 and a display adapter 211. Keyboard 212,mouse 213 and speaker 214 may all be interconnected to bus 202 throughuser interface adapter 210. A display monitor 215 may be connected tosystem bus 202 by display adapter 211. In this manner, a user is capableof inputting to predictive analytics system 101 through keyboard 212 ormouse 213 and receiving output from predictive analytics system 101 viadisplay 215 or speaker 214. Other input mechanisms may be used to inputdata to predictive analytics system 101 that are not shown in FIG. 2,such as display 215 having touch-screen capability and keyboard 212being a virtual keyboard. Predictive analytics system 101 of FIG. 2 isnot to be limited in scope to the elements depicted in FIG. 2 and mayinclude fewer or additional elements than depicted in FIG. 2.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a computer, or other programmable data processing apparatusto produce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks. These computerreadable program instructions may also be stored in a computer readablestorage medium that can direct a computer, a programmable dataprocessing apparatus, and/or other devices to function in a particularmanner, such that the computer readable storage medium havinginstructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be accomplished as one step, executed concurrently,substantially concurrently, in a partially or wholly temporallyoverlapping manner, or the blocks may sometimes be executed in thereverse order, depending upon the functionality involved. It will alsobe noted that each block of the block diagrams and/or flowchartillustration, and combinations of blocks in the block diagrams and/orflowchart illustration, can be implemented by special purposehardware-based systems that perform the specified functions or acts orcarry out combinations of special purpose hardware and computerinstructions.

As stated in the Background section, achieving better predictive modelsis an objective in the research and practice of machine learningtechniques. For example, ensemble methods use multiple learningalgorithms to obtain better predictive performance than could beobtained from any of the constituent learning algorithms alone. Suchensemble methods include bootstrap aggregating (also called bagging),boosting, etc. Bootstrap aggregating is a machine learning ensemblemeta-algorithm designed to improve the stability and accuracy of machinelearning algorithms used in statistical classification and regression.Boosting is a machine learning ensemble meta-algorithm for primarilyreducing bias and also variance in supervised learning and a family ofmachine learning algorithms that convert weak learners to strong ones.In such techniques, such as boosting, the weights of wrongly classifiedcases are increased while the weights of correctly classified cases aredecreased during the modeling process. Such a strategy (heuristic) doesachieve better predictions in many cases; however, overfittingoutliers/noises is a possibility. As a result of overfittingoutliers/noises, the predictive accuracy is lessened. Hence, theheuristic strategy of increasing the weights of wrongly classified casesand decreasing the weights of correctly classified cases may not be thebest strategy for improving the prediction accuracy of the model. Forexample, sometimes, it may be better to increase the weights ofcorrectly classified cases because such cases contain very importantpatterns which should be learned by the machine learning algorithm. Itmay also be better to decrease the weights of wrongly classified cases,such as outlier cases, for similar reasons. Consequently, suchtechniques, such as boosting, fail to identify the optimal weights forthe classified cases, and therefore, fail to achieve optimal predictionaccuracy in machine learning techniques.

The embodiments of the present invention provide a means for achievingoptimal prediction accuracy in machine learning techniques byidentifying the optimal weights using the concept of learning to teachinvolving two intelligent agents (a teacher model and a student model)as discussed below in connection with FIGS. 3-6. FIG. 3 is a diagram ofthe software components of predictive analytics system 101 used toidentify the optimal weights to be used by the predictive model forgenerating predictions. FIG. 4 illustrates the state features generatedby the student model. FIG. 5 illustrates the rewards generated by thereward generator. FIG. 6 is a flowchart of a method for improvingprediction accuracy in machine learning techniques.

Given a training data {<X_k, f_k, y_k>|k=1, . . . , N}, where X_k is avector of predictors in case k, y_k is the observed target, and f_k isan optional case weight (let f_k=1 if no case weight exists), thepresent invention will generate optimal case weights f_k^(opt) for eachcase k. With the optimal case weights, a student model will be trainedto provide accurate predictions using the held-out dataset as discussedbelow.

As stated above, FIG. 3 is a diagram of the software components ofpredictive analytics system 101 used to identify the optimal weights tobe used by the predictive model for generating predictions in accordancewith an embodiment of the present invention. In one embodiment, thesesoftware components may reside in application 204 (FIG. 2) of predictiveanalytics system 101.

The following provides a brief description of these software components.A more detailed description of these software components (includingtheir functionalities) is provided below in conjunction with FIGS. 4-6.

Referring to FIG. 3, in conjunction with FIGS. 1-2, predictive analyticssystem 101 includes a module referred to herein as the “teacher model”301. In one embodiment, teacher model 301 is a neural network configuredto receive state features as inputs and generate a weight for each datacase. In one embodiment, the weight parameters of the neural network areinitialized randomly before the training process.

Predictive analytics system 101 further includes a module referred toherein as the “current student model 302,” which receives the weightsgenerated by teacher model 301 and generates state features, such asthose shown in FIG. 4. A “student model,” as used herein, refers to astatistical model that is trained to provide a prediction using trainingdata 103. A “current” student model 302, as used herein, refers to astudent model currently being trained by predictive analytics system 101to provide a prediction using training data 103. In one embodiment,student model 302 corresponds to a learner, such as a decision tree or aneural network. In one embodiment, decision tree learning uses adecision tree as a predictive model to go from observations about anitem (represented in the branches) to conclusions about the items'target value (represented in the leaves). In one embodiment, a neuralnetwork is a network or circuit of neurons (artificial neurons) ornodes.

FIG. 4 illustrates the state features generated by student model 302 inaccordance with an embodiment of the present invention.

Referring to FIG. 4, such state features may include predictors 401,their corresponding weight 402 and their predicted value 403. In oneembodiment, predictor 401 corresponds to the outcome variable, such asthe observed target. Weight 402 corresponds to the weight assigned tosuch a predictor obtained from teacher model 301. In one embodiment,such weights 402 indicate the confidence that the correspondingpredicted value 403 of predictor 401 is accurate. The higher the value,the greater confidence in the corresponding predicted value 403 beingcorrect.

In one embodiment, such state features may also include data features,case weights, student model features and features to represent thecombination of both the data and the student model.

In one embodiment, such state features may be utilized by teaching model301 to generate new weights which are used to train a candidate studentmodel 303. A “candidate student model 303,” as used herein, refers to astudent model that is being examined to determine if is a better studentmodel (better at predicting the observed target) than the currentstudent model.

In one embodiment, candidate student model 303 is trained by usingtraining data 103 and the new weights generated by teacher model 301.

As illustrated in FIG. 3, a module, referred to herein as the “rewardgenerator 304,” generates rewards by comparing the current and candidatestudent models 302, 303 using training data 103 and testing data 104(“held-out data”). In one embodiment, teacher model 301 is updated withthe rewards. In one embodiment, teacher model 301 utilizes the rewardsto update its parameters via policy gradient methods, such asreinforcement learning.

A “reward,” as used herein, refers to a value generated by a function(reward function) used in reinforcement learning. The goal of areinforcement learning agent (predictive analytics system 101) is tocollect as much reward as possible. In one embodiment, a positive rewardis returned by reward generator 304 if the candidate student model 303is better at predicting the observed target than the current studentmodel 302. Conversely, a negative reward is returned by reward generator304 if the current student model 302 is better at predicting theobserved target than the candidate student model 303. In one embodiment,reward generator 304 is part of candidate student model 303.

In one embodiment, such rewards are generated by reward generator 304 byapplying training data 103, testing data 104 to student models 302, 303as shown in FIG. 5.

FIG. 5 illustrates the rewards generated by reward generator 304 inaccordance with an embodiment of the present invention.

Referring to FIG. 5, reward generator 304 generates a model level reward501 and a case level reward 502. In one embodiment, model level reward501 refers to the reward associated with the student models 302, 303generating a prediction for the observed target based on the testingdata 104. For instance, reward 501 is generated based on how much bettercandidate student model 303 is at predicting the observed target thanthe current student model 302 using testing data 104. A positive reward501 is returned by reward generator 304 if the candidate student model303 is better at predicting the observed target than the current studentmodel 302 using testing data 104. Conversely, a negative reward 501 isreturned by reward generator 304 if the current student model 302 isbetter at predicting the observed target than the candidate studentmodel 303 using testing data 104.

Case level reward 502 refers to the reward based on correctlyclassifying the data case by student models 302, 303 using training data103. If student model 302, 303 correctly classified the data case, thena positive reward 502 is returned by reward generator 304. Conversely, anegative reward 502 is returned by reward generator 304 if student model302, 303 did not correctly classify the data case.

In one embodiment, reward generator 304 generates a final reward 503that is a combination of model level reward 501 and case level reward502, such an average of the two rewards.

Returning to FIG. 3, utilizing the rewards, a module, referred to hereinas the “updater 305,” updates teacher model 301 with the reward. In oneembodiment, teacher model 301 faces an associativeimmediate-reinforcement learning task. Suppose that the reward for thereinforcement value is r at each trial, then the parameter w_ij in thenetwork is incremented by an amount Δw_ij=Σ_(k=1){circumflex over ( )}N

α(∂lng_k)/(∂w_ij)r

, where a is a learning rate factor, N is a positive integer number, andg_k is the output of the teacher model for case k.

Once teacher model 301 is updated, a determination is made by decisionmaker 306 as to whether candidate student model 303 is a betterpredictor of the observed target than the current student model 302 ornot. In one embodiment, if candidate student model 303 is better atpredicting the observed target than current student model 302, thendecision maker 306 will update current student model 302 with candidatestudent model 303 as well as update the current weights with the newweights. The updated student model 302 will then generate new statefeatures which are inputted to teacher model 301.

Alternatively, if candidate student model 303 is not better atpredicting the observed target than current student model 302, thendecision maker 306 directly requests the updated teacher model 301(updated with the rewards as discussed above) to generate new weightsusing the current student features from the current student model 302.

A more detailed discussion regarding the process of improving predictionaccuracy in machine learning techniques using the software componentsdiscussed above is provided below in connection with FIG. 6.

FIG. 6 is a flowchart of a method 600 for improving prediction accuracyin machine learning techniques in accordance with an embodiment of thepresent invention.

Referring to FIG. 6, in conjunction with FIGS. 1-5, in step 601,predictive analytics system 101 constructs a teacher model 301 whichgenerates a weight for each data case. A “teacher model,” as usedherein, refers to a statistical model that determines the appropriatedata, loss function and hypothesis space to facilitate the learning ofthe student model. In one embodiment, teacher model 301 is a neuralnetwork. In one embodiment, teacher model 301 receives state featuresfrom current student model 302 as inputs and generates a weight for eachdata case. A “data case,” as used herein, refers to data used to predictan observed target. In one embodiment, the weight parameters areinitialized randomly before the training process begins (discussedfurther below).

In one embodiment, teacher model 301 includes networks composed ofseveral layers. In one embodiment, the layers are made of nodes, where anode is a place where computation happens, loosely patterned on a neuronin the human brain, which fires when it encounters sufficient stimuli. Anode combines input from the data, such as state features (discussedfurther below) from current student model 302 with a set ofcoefficients, or weights, that either amplify or dampen that input,thereby assigning significance to inputs with regard to the task thealgorithm is trying to learn (e.g., which input is most helpful inclassifying data without error). These input-weight products are summedand then the sum is passed through a node's so-called activationfunction, to determine whether and to what extent that signal shouldprogress further through the network to affect the ultimate outcome(e.g., an act of classification). If the signals pass through, theneuron has been “activated.”

In one embodiment, a node layer is a row of neuron-like switches thatturn on or off as the input is fed through the net. Each layer's outputis simultaneously the subsequent layer's input, starting from an initialinput layer receiving the data.

In one embodiment, the model's adjustable weights are paired with theinput features so as to assign significance to those features withregard to how the neural network classifies and clusters input.

In one embodiment, such generated weights can be used to correct thedistribution of data cases in training data 103 in the event thattraining data 103 comes from a biased sampling.

In step 602, predictive analytics system 101 trains current studentmodel 302 using training data 103 and weights (current weight for eachdata case) generated by teacher model 301. As previously discussed,current student model 302 refers to a student model (statistical modelthat is trained to provide a prediction, such as predicting the observedtarget, using training data) that is currently being trained bypredictive analytics system 101 to provide a prediction using trainingdata 103. As also previously discussed, in one embodiment, currentstudent model 302 corresponds to a learner, such as a decision tree or aneural network. In one embodiment, decision tree learning uses adecision tree as a predictive model to go from observations about anitem (represented in the branches) to conclusions about the items'target value (represented in the leaves). In one embodiment, a neuralnetwork is a network or circuit of neurons (artificial neurons) ornodes.

In one embodiment, case weights are used as actions on current studentmodel 302 (as well as on candidate student model 303 discussed furtherbelow). Such a method allows the usage of any type of machine learner asthe student model (student models 302, 303) given that the learnersupports case weight in training.

In one embodiment, such training involves fitting current student model302 on a training dataset 103, that is a set of examples used to fit theparameters, such as the weights generated by teacher model 301. In oneembodiment, current student model 302 is trained using a supervisedlearning method (e.g., gradient descent or stochastic gradient descent).In one embodiment, training dataset 103 consists of pairs of inputvectors (or scalar) and the corresponding output vector (or scalar),which may be denoted as the target. Current student model 302 is runwith training dataset 103 and produces a result, which is then comparedwith the target, for each input vector in training dataset 103. Based onthe result of the comparison and the specific learning algorithm beingused, the parameters of student model 302 are adjusted.

In step 603, after training current student model 302, the trainedcurrent student model 302 generates state features. In one embodiment,state features are defined for each data case based on current studentmodel 302. State features may include, but not limited to, data featurescontaining information for a data case, such as its predictors, target,etc.; case weight generated by teacher model 301; student modelfeatures, including the measures reflecting how well current studentmodel 302 is trained; and features to represent the combination of bothdata and student model 302, such as predicted targets, probabilities ofeach target category, etc.

In one embodiment, case-level state features are also generated bystudent model 302, including the corresponding case weight.

In one embodiment, current student model 302 generates data thatincludes the previously discussed state features. Such features aregenerated using a set of statistical assumptions based on the receivingtraining data 103 and weights from teacher model 301.

In step 604, teacher model 301 generates new weights using the statefeatures generated by current student model 302. In one embodiment, suchstate features are input to teacher model 301 which are used to generateweights by teacher model 301 as discussed above.

In step 605, predictive analytics system 101 trains a candidate studentmodel 303 using training data 103 and the new weights generated byteacher model 301. In one embodiment, training candidate student model303 is performed in the same manner as training current student model302 as discussed above in connection with step 602.

In step 606, reward generator 304 generates a reward by comparing thecurrent and candidate student models 302, 303 (after training studentmodels 302, 303) using training data 103 and testing data 104 (“held-outdata”) to determine which is better at predicting an observed target. A“reward,” as used herein, refers to a value generated by a function(reward function) used in reinforcement learning. The goal of areinforcement learning agent (predictive analytics system 101) is tocollect as much reward as possible. In one embodiment, a positive modellevel reward 501 is returned by reward generator 304 if the candidatestudent model 303 is better at predicting the observed target than thecurrent student model 302 using testing data 104. Conversely, a negativemodel level reward 501 is returned by reward generator 304 if thecurrent student model 302 is better at predicting the observed targetthan the candidate student model 303 using testing data 104.

Case level reward 502 refers to the reward based on correctlyclassifying the data case by student models 302, 303 using training data103. If student model 302, 303 correctly classified the data case, thena positive reward 502 is returned by reward generator 304. Conversely, anegative reward 502 is returned by reward generator 304 if student model302, 303 did not correctly classify the data case.

In one embodiment, reward generator 304 generates a final reward 503that is a combination of model level reward 501 and case level reward502, such an average of the two rewards.

In step 607, a determination is made by predictive analytics system 101as to whether a stopping rule has been satisfied. “Stopping rules,” asused herein, refer to the rules that determine whether the training of astudent model has been completed. Such training is completed when it hasbeen determined that the optimal weights for the data cases have beenidentified.

In one embodiment, such stopping rules include, but not limited to, thefollowing: reaching a specified number of trials, reaching a specifiedtraining timing, converging of a prediction accuracy and auser-initiated termination.

If any of these stopping rules have been satisfied, then, in step 608,the optimal weights (weights generated by teacher model 301) and thecorresponding student model 302 are returned to the user, such as via auser interface on predictive analytics system 101. The returned studentmodel is able to provide an accurate prediction of the observed target.Furthermore, the returned weights, which are generated by teacher model301 (see step 604), are deemed to be the “optimal” weights, such as whena stopping rule has been satisfied (e.g., when the prediction accuracyconverges). In this manner, optimal weights are identified to improveprediction accuracy.

If, however, none of the stopping rules have been satisfied, then, instep 609, updater 305 updates teacher model 301 with the reward (rewardof step 606). In one embodiment, teacher model 301 utilizes the rewardsto update its parameters via policy gradient methods, such asreinforcement learning.

As previously discussed, in one embodiment, teacher model 301 faces anassociative immediate-reinforcement learning task. Suppose that thereward for the reinforcement value is r at each trial, then theparameter w_ij in the network is incremented by an amountΔw_ij=Σ(k=1){circumflex over ( )}N

α(∂lng_k)/(θw_ij)r

, where α is a learning rate factor, N is a positive integer number, andg_k is the output of the teacher model for case k. The incrementedamount Δw upon which teacher model 301 will be updated may also be shownas follows:

${\Delta w_{ij}} = {\alpha r^{model}{\sum\limits_{k = 1}^{N}{\frac{{\partial\ln}\; g_{k}}{\partial w_{ij}}r_{k}^{case}}}}$

where r is the reinforcement value at each trial, w_(ij) is a parameterin the network incremented by an amount Δw_(ij), α is a learning ratefactor, N is a positive integer number, and g_(k) is the output of theteacher model for case k.

Upon updating teacher model 301, in step 610, a determination is made bydecision maker 306 as to whether candidate student model 303 generates abetter prediction of the observed target than current student model 302.Such a determination is based on how close the prediction is to theobserved target.

If candidate student model 303 is better at predicting the observedtarget than current student model 302, then, in step 611, currentstudent model 302 is updated with candidate student model 303. That is,in step 611, current student model 302 is essentially replaced withcandidate student model 303.

Furthermore, if candidate student model 303 is better at predicting theobserved target than current student model 302, then in step 612, thecurrent weights are updated with the new weights (new weights generatedby teacher model 301 in step 604).

Upon updating the current weights with the new weights, the updatedstudent model 302 (updated in step 611) generates state features in step603 using the new weights.

Alternatively, if candidate student model 303 is not better atpredicting the observed target than current student model 302, thendecision maker 306 directly requests the updated teacher model 301(updated in step 609) to generate new weights in step 604 using thecurrent student features from the current student model 302 and currentweights.

In this manner, the present invention devises a framework whichimplements the concept of “learning to teach” in the field of predictivemodeling. Such a framework includes a teacher model, which generates aweight for each data case. The training data cases, along with thegenerated weights, are used to re-train the student model. A reward isreturned by evaluating the trained student model on a held-out datasetin terms of prediction accuracy. The teacher model then utilizes thereward to update its parameters via policy gradient methods, e.g.,reinforcement learning. Such a process will be repeated until thestudent model achieves desired performance.

In comparison to previously used heuristic methods (e.g., boosting), theapproach of the present invention determines case weights in an optimalway. This allows one to build a better student model via basic learners,e.g., decision tree, neural network, etc., rather than using an ensemblemodel.

By using case weights as actions on the student model, any type ofmachine learner may be used as the student model given that the learnersupports case weight in training.

Furthermore, the present invention improves the technology or technicalfield involving predictive modeling. As discussed above, achievingbetter predictive models is an objective in the research and practice ofmachine learning techniques. For example, ensemble methods use multiplelearning algorithms to obtain better predictive performance than couldbe obtained from any of the constituent learning algorithms alone. Suchensemble methods include bootstrap aggregating (also called bagging),boosting, etc. Bootstrap aggregating is a machine learning ensemblemeta-algorithm designed to improve the stability and accuracy of machinelearning algorithms used in statistical classification and regression.Boosting is a machine learning ensemble meta-algorithm for primarilyreducing bias and also variance in supervised learning and a family ofmachine learning algorithms that convert weak learners to strong ones.In such techniques, such as boosting, the weights of wrongly classifiedcases are increased while the weights of correctly classified cases aredecreased during the modeling process. Such a strategy (heuristic) doesachieve better predictions in many cases; however, overfittingoutliers/noises is a possibility. As a result of overfittingoutliers/noises, the predictive accuracy is lessened. Hence, theheuristic strategy of increasing the weights of wrongly classified casesand decreasing the weights of correctly classified cases may not be thebest strategy for improving the prediction accuracy of the model. Forexample, sometimes, it may be better to increase the weights ofcorrectly classified cases because such cases contain very importantpatterns which should be learned by the machine learning algorithm. Itmay also be better to decrease the weights of wrongly classified cases,such as outlier cases, for similar reasons. Consequently, suchtechniques, such as boosting, fail to identify the optimal weights forthe classified cases, and therefore, fail to achieve optimal predictionaccuracy in machine learning techniques.

The present invention improves such technology by constructing a teachermodel, where the teacher model generates a weight for each data case. A“teacher model,” as used herein, refers to a statistical model thatdetermines the appropriate data, loss function and hypothesis space tofacilitate the learning of the student model. The current student modelis then trained using training data and the weights generated by theteacher model. A “student model,” as used herein, refers to astatistical model that is trained to provide a prediction using trainingdata. A “current” student model, as used herein, refers to a studentmodel currently being trained to provide a prediction using trainingdata. The current student model generates state features (e.g., datafeatures, case weights, student model features and features to representthe combination of both the data and the student model), which are usedby the teacher model to generate new weights. A candidate student modelis then trained using training data and these new weights. A “candidatestudent model,” as used herein, refers to a student model that is beingexamined to determine if is a better student model (better at predictingthe observed target) than the current student model. A reward is thengenerated by comparing the current student model with the candidatestudent model using training and testing data to determine which isbetter at predicting an observed target. A “reward,” as used herein,refers to a value generated by a function (reward function) used inreinforcement learning. A positive reward may be returned if thecandidate student model is better at predicting the observed target thanthe current student model. Conversely, a negative reward may be returnedif the current student model is better at predicting the observed targetthan the candidate student model. The teacher model is then updated withthe reward. The teacher model utilizes the rewards to update itsparameters via policy gradient methods, such as reinforcement learning.If the candidate student model is better at predicting the observedtarget than the current student model, then the current student model isupdated with the candidate student model and the current weights areupdated with the new weights generated by the teacher model. Uponupdating the current weights with the new weights, the current studentmodel generates new state features. If, however, the candidate studentmodel is not better at predicting the observed target than the currentstudent model, then the updated teacher model generates new weightsusing the current weights and the current student features from thecurrent student model. Upon any of the stopping rules being satisfied(e.g., reaching a specified number of trials, reaching a specifiedtraining timing, converging of a prediction accuracy and auser-initiated termination), the weights generated by the teacher modelare deemed to be the “optimal” weights which are returned to the useralong with the corresponding student model. In this manner, optimalweights are identified to improve prediction accuracy. Furthermore, inthis manner, there is an improvement in the technical field ofpredictive modeling.

The technical solution provided by the present invention cannot beperformed in the human mind or by a human using a pen and paper. Thatis, the technical solution provided by the present invention could notbe accomplished in the human mind or by a human using a pen and paper inany reasonable amount of time and with any reasonable expectation ofaccuracy without the use of a computer.

In one embodiment of the present invention, a computer-implementedmethod for improving prediction accuracy in machine learning techniquescomprises training a current student model using training data andweights generated by a teacher model. The method further comprisesgenerating new weights by the teacher model using state features. Themethod additionally comprises training a candidate student model usingthe training data and the new weights. Furthermore, the method comprisesgenerating a reward by comparing the current student model with thecandidate student model using the training data and testing data todetermine which is better at predicting an observed target.Additionally, the method comprises returning the new weights and thecurrent student model to a user in response to a stopping rule beingsatisfied, where the returned student model provides a prediction of theobserved target.

In one embodiment of the present invention, the method further comprisesdetermining whether the candidate student model generates a betterprediction of the observed target than the current student model basedon how close the prediction is to the observed target.

In one embodiment, the method further comprises updating the currentstudent model with the candidate student model and updating currentweights with the new weights in response to the candidate student modelgenerating a better prediction of the observed target than the currentstudent model.

In one embodiment, the method additionally comprises updating theteacher model with the reward in response to a stopping rule not beingsatisfied.

Furthermore, in one embodiment, the method additionally comprisesgenerating a second set of new weights by the updated teacher modelusing the state features in response to the candidate student model notgenerating a better prediction of the observed target than the currentstudent model.

Additionally, in one embodiment, the method further comprises trainingthe candidate student model using the training data and the second setof new weights.

In one embodiment, the method further comprises having the stopping rulecomprise one or more of the following: reaching a specified number oftrials, reaching a specified training time, converging of a predictionaccuracy, and a user-initiated termination.

Other forms of the embodiments of the method described above are in asystem and in a computer program product.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

1. A computer-implemented method for improving prediction accuracy inmachine learning techniques, the method comprising: training a currentstudent model using training data and weights generated by a teachermodel; generating new weights by said teacher model using statefeatures; training a candidate student model using said training dataand said new weights; generating a reward by comparing said currentstudent model with said candidate student model using said training dataand testing data to determine which is better at predicting an observedtarget; and returning said new weights and said current student model toa user in response to a stopping rule being satisfied, wherein saidreturned student model provides a prediction of said observed target. 2.The method as recited in claim 1 further comprising: determining whethersaid candidate student model generates a better prediction of saidobserved target than said current student model based on how close theprediction is to said observed target.
 3. The method as recited in claim2 further comprising: updating said current student model with saidcandidate student model and updating current weights with said newweights in response to said candidate student model generating a betterprediction of said observed target than said current student model. 4.The method as recited in claim 1 further comprising: updating saidteacher model with said reward in response to a stopping rule not beingsatisfied.
 5. The method as recited in claim 4 further comprising:generating a second set of new weights by said updated teacher modelusing said state features in response to said candidate student modelnot generating a better prediction of said observed target than saidcurrent student model.
 6. The method as recited in claim 5 furthercomprising: training said candidate student model using said trainingdata and said second set of new weights.
 7. The method as recited inclaim 1, wherein said stopping rule comprises one or more of thefollowing: reaching a specified number of trials, reaching a specifiedtraining time, converging of a prediction accuracy, and a user-initiatedtermination.
 8. A computer program product for improving predictionaccuracy in machine learning techniques, the computer program productcomprising one or more computer readable storage mediums having programcode embodied therewith, the program code comprising programminginstructions for: training a current student model using training dataand weights generated by a teacher model; generating new weights by saidteacher model using state features; training a candidate student modelusing said training data and said new weights; generating a reward bycomparing said current student model with said candidate student modelusing said training data and testing data to determine which is betterat predicting an observed target; and returning said new weights andsaid current student model to a user in response to a stopping rulebeing satisfied, wherein said returned student model provides aprediction of said observed target.
 9. The computer program product asrecited in claim 8, wherein the program code further comprises theprogramming instructions for: determining whether said candidate studentmodel generates a better prediction of said observed target than saidcurrent student model based on how close the prediction is to saidobserved target.
 10. The computer program product as recited in claim 9,wherein the program code further comprises the programming instructionsfor: updating said current student model with said candidate studentmodel and updating current weights with said new weights in response tosaid candidate student model generating a better prediction of saidobserved target than said current student model.
 11. The computerprogram product as recited in claim 8, wherein the program code furthercomprises the programming instructions for: updating said teacher modelwith said reward in response to a stopping rule not being satisfied. 12.The computer program product as recited in claim 11, wherein the programcode further comprises the programming instructions for: generating asecond set of new weights by said updated teacher model using said statefeatures in response to said candidate student model not generating abetter prediction of said observed target than said current studentmodel.
 13. The computer program product as recited in claim 12, whereinthe program code further comprises the programming instructions for:training said candidate student model using said training data and saidsecond set of new weights.
 14. The computer program product as recitedin claim 8, wherein said stopping rule comprises one or more of thefollowing: reaching a specified number of trials, reaching a specifiedtraining time, converging of a prediction accuracy, and a user-initiatedtermination.
 15. A system, comprising: a memory for storing a computerprogram for improving prediction accuracy in machine learningtechniques; and a processor connected to said memory, wherein saidprocessor is configured to execute program instructions of the computerprogram comprising: training a current student model using training dataand weights generated by a teacher model; generating new weights by saidteacher model using state features; training a candidate student modelusing said training data and said new weights; generating a reward bycomparing said current student model with said candidate student modelusing said training data and testing data to determine which is betterat predicting an observed target; and returning said new weights andsaid current student model to a user in response to a stopping rulebeing satisfied, wherein said returned student model provides aprediction of said observed target.
 16. The system as recited in claim15, wherein the program instructions of the computer program furthercomprise: determining whether said candidate student model generates abetter prediction of said observed target than said current studentmodel based on how close the prediction is to said observed target. 17.The system as recited in claim 16, wherein the program instructions ofthe computer program further comprise: updating said current studentmodel with said candidate student model and updating current weightswith said new weights in response to said candidate student modelgenerating a better prediction of said observed target than said currentstudent model.
 18. The system as recited in claim 15, wherein theprogram instructions of the computer program further comprise: updatingsaid teacher model with said reward in response to a stopping rule notbeing satisfied.
 19. The system as recited in claim 18, wherein theprogram instructions of the computer program further comprise:generating a second set of new weights by said updated teacher modelusing said state features in response to said candidate student modelnot generating a better prediction of said observed target than saidcurrent student model.
 20. The system as recited in claim 19, whereinthe program instructions of the computer program further comprise:training said candidate student model using said training data and saidsecond set of new weights.